1. Field of the Invention
The present invention relates to an image processing device, an image processing method, and an imaging apparatus.
Priority is claimed on Japanese Patent Application No. 2013-207603, filed Oct. 2, 2013, the content of which is incorporated herein by reference.
2. Description of Related Art
In recent years, with high functionalization of imaging apparatuses such as digital still cameras or video cameras, image processing on image data of photographed images has become complicated. In processing flows of general imaging apparatuses, an imaging processing unit first performs an imaging process on a pixel signal of a photographed image captured by an imaging element and temporarily stores the image data subjected to the imaging process in a memory. Subsequently, an image processing unit reads the image data subjected to the imaging process and stored in the memory, performs various kinds of image processing such as an image-quality improvement process, and stores the image data subjected to the image processing in the memory again. Thereafter, for example, when the image data subjected to the image processing is displayed on a display device, a display processing unit reads the image data stored in the memory, performs a display conversion process, and displays the image data on the display device.
In a series of processing flows of the imaging apparatuses, content of the image processing performed on the image data subjected to the imaging process by the image processing unit differs according to setting of an operation mode or the like of the imaging apparatuses. For example, in image processing for generating a contracted image corresponding to a photographed image, i.e., a thumbnail image, calculation processes such as an OB (optical black)/WB (white balance) process, three-plate processing, a noise reduction process, lowpass filter (LPF) processing, a contraction process, and a color conversion process are sequentially performed to generate a thumbnail image. For example, in image processing for generating a partially expanded image partially expanded from a photographed image, i.e., a digital zoom image, calculation processes such as an OB/WB process, three-plate processing, a noise reduction process, an expansion process, and an edge enhancement process are sequentially performed to generate a digital zoom image.
Thus, in the image processing unit included in the imaging apparatus, content of the image processing or an order in which the image processing is performed on the image data subjected to the imaging process, i.e., a calculation process, is different for each image processing to be performed. Therefore, in the image processing unit, a processing form is realized such that each calculation unit blocked for each piece of content of the calculation process to be performed is prepared, a pipeline in which the plurality of calculation units are connected in series is configured, and a series of various kinds of image processing is realized. In the processing form of the pipeline, the calculation units perform the calculation process in parallel in such a manner that the calculation units on the front stages perform the calculation process on the image data in each pixel or each of the regions divided from an image and sequentially deliver the image data subjected to the calculation process to the calculation units on the rear stages.
In the image processing unit, various different kinds of image processing are dealt with according an operation mode or the like of the imaging apparatus by transposing the calculation units forming the pipeline and the order of the calculation processes performed in the pipeline. Therefore, the image processing unit is configured such that a combination of the calculation units forming the pipeline or the order in which the calculation units are disposed on the pipeline can be transposed freely according to various kinds of image processing to be performed. At this time, in the image processing unit, it is necessary to synchronize delivery of the image data between the calculation units forming the pipeline.
For example, the technology disclosed in Japanese Unexamined Patent Application, First Publication No. H10-334225 discloses an image processing apparatus configured such that buffers receiving image data output from front processing stages and transferring the received image data to rear processing stages are disposed between the processing stages corresponding to calculation units that perform a calculation process. In the image processing apparatus disclosed in Japanese Unexamined Patent Application, First Publication No. H10-334225, a double buffer is configured by two buffers, and a control device controls changeover of the double buffer and starting of a process performed on the image data by each processing stage. Further, in the image processing apparatus disclosed in Japanese Unexamined Patent Application, First Publication No. H10-334225, a series of processes of a pipeline is realized in such a manner that the image data processed by the processing stages is sequentially delivered via the double buffer so that the image data processed by a front processing stage is sent to a subsequent processing stage via the double buffer and the image data processed by the subsequent processing stage is sent to a further subsequent processing stage via the double buffer. Furthermore, in the image processing apparatus disclosed in Japanese Unexamined Patent Application, First Publication No. H10-334225, the processing stages sequentially output the image data, so that the processing stages operate simultaneously in parallel.
In the series of image processing for which the pipeline is configured, a processing amount of the calculation process to be performed by each calculation unit is different according to the data amount of a calculation target image data. The data amount of a calculation target image data is different according to, for example, the size of an image to be subjected to a calculation process or the size of a region in the image to be subjected to a calculation process. Therefore, for example, in the calculation unit of which a processing amount is small, there is a non-processing time called a waiting time, an idle time, a standby time, a slack time, or the like in which the image data is not actually processed.